Distribution-Free Learning of Bayesian Network Structure in Continuous Domains

In this paper we present a method for learning the structure of Bayesian networks (BNs) without making any assumptions on the probability distribution of the domain. This is mainly useful for continuous domains, where there is little guidance and many choices for the parametric distribution families to be used for the local conditional probabilities of the Bayesian network, and only a few have been examined analytically. We therefore focus on BN structure learning in continuous domains. We address the problem by developing a conditional independence test for continuous variables, which can be readily used by any existing independence-based BN structure learning algorithm. Our test is non-parametric, making no assumptions on the distribution of the domain. We also provide an effective and computationally efficient method for calculating it from data. We demonstrate the learning of the structure of graphical models in continuous domains from real-world data, to our knowledge for the first time using independence-based methods and without distributional assumptions. We also experimentally show that our test compares favorably with existing statistical approaches which use prediscretization, and verify desirable properties such as statistical consistency. Introduction and Motivation One of the first problems that a researcher who is interested in learning a graphical model from data is faced with, is making a choice on the kind of probability distributions she will use. Such distributions are used to model local interactions among subsets of variables in the model. For example, in a Bayesian network (BN), a local probability distribution function (PDF) needs to be defined between every variable and its parents. The choice is easier in discrete domains, where every variable can take only a fixed number of values; the standard choice for discrete PDFs is the multinomial, which is usually sufficient for modeling complex interactions among variables, and whose parameters are straightforward to estimate from data. In continuous or mixed continuous-discrete domains however the problem is considerably harder, prompting the use of simplifying assumptions. The common assumption is for the local PDFs between parents and children to be linear Copyright c © 2005, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. relations with additive Gaussian errors (Geiger & Heckerman 1994; Spirtes, Glymour, & Scheines 1993). However, there are many real-life situations where this assumption fails (e.g., stock market prices, biometric variables, weather status, etc.), where the interactions are far from linear. In these cases, such an assumption can lead to inaccurate networks that are a poor fit to the data and the underlying probability distribution, and can produce incorrect structures. Our work addresses the learning of the structure of graphical models without making any such assumptions about the distributions of the local PDFs. Although we focus on Bayesian network structure learning, application to Markov network structure learning is straightforward. To learn the structure of a Bayesian network, there exist two general classes of algorithms. The first, called the score-based approach, employs a search in the space of all possible legal structures guided by a heuristic function, usually penalized log-likelihood (Lam & Bacchus 1994). The search procedure maximizes this score, usually by hill-climbing. Other search techniques have also been used (Heckerman 1995). The second class of Bayesian network structure learning algorithms use the fact that the structure of a BN implies that a set of conditional independence statements hold in the domain it is modeling. They exploit this property by conducting a number of statistical conditional independence (CI) tests on the data and use their results to make inferences about the structure. Assuming no errors in these tests, the idea is to constrain, if possible, the set of possible structures that satisfy the conditional independencies that are found in the data to a singleton, and infer that structure as the only possible one. For this reason these algorithms are called constraint-based or independence-based. Our work uses the second class of algorithms to learn BN structure in continuous domains. The crucial observation is that the independence-based algorithms interact with the data only through the conditional independence tests done during their operation. Therefore, the use of a CI test between continuous variables, and in particular one that does not assume anything about the distribution of the variables involved in the test, would be sufficient for learning the structure in such domains in a distribution-independent fashion. CI tests that do not assume any particular family of distributions are called non-parametric. Although such tests exist for discrete variables—the χ (chi-square) test of indeFigure 1: Two very different data sets have similar 3 × 3 histograms. Left: X, Y strongly dependent. Right: X, Y independent by construction. pendence is perhaps the most common one—for continuous variables the standard approach is to discretize the continuous variables and perform a discrete test. One problem with this method is that the discretization has to be done with care; for example, Fig. 1 depicts two very different situations where X and Y are dependent (left) and independent (right), that produce two very similar histograms after a 3×3 discretization. The (unconditional) multi-resolution test of Margaritis & Thrun (2001), outlined in the next section, addresses cases such as this. Independence-based algorithms however require conditional independence tests. In this paper we extend the above-mentioned unconditional test to a conditional version that carefully discretizes the Z axis, performs an (unconditional) independence test on the data in each resulting bin, and combines the results into a single probabilistic measure of conditional independence. Outline of Unconditional Test In this section we outline the unconditional multi-resolution test of independence between two continuous variables X and Y of Margaritis & Thrun (2001). A central idea is the comparison of two competing statistical models, MI (the independent model) and M¬I (the dependent model), according to the data likelihood of a data set consisting of (X, Y ) pairs. For a given fixed resolution, the test uses a discretized version of the data set at that resolution (resolution is the size of the histogram or “grid” placed over the data e.g., 3×3 in Fig. 1). The dependent model M¬I corresponds to a joint multinomial distribution while the independent model MI to two marginal multinomials along the Xand Y -axes. Margaritis & Thrun calculate the data likelihoods of each model analytically: